Skip to content

[MLOB-2411] Add Distributed Proxy/Gateway Service Guide for LLM Observability #28593

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 19 commits into from
Jun 9, 2025
Merged

[MLOB-2411] Add Distributed Proxy/Gateway Service Guide for LLM Observability #28593

merged 19 commits into from
Jun 9, 2025

Conversation

sabrenner
Copy link
Contributor

What does this PR do? What is the motivation?

Merge instructions

Merge readiness:

  • Ready for merge

For Datadog employees:
Merge queue is enabled in this repo. Your branch name MUST follow the <name>/<description> convention and include the forward slash (/). Without this format, your pull request will not pass in CI, the GitLab pipeline will not run, and you won't get a branch preview. Getting a branch preview makes it easier for us to check any issues with your PR, such as broken links.

If your branch doesn't follow this format, rename it or create a new branch and PR.

To have your PR automatically merged after it receives the required reviews, add the following PR comment:

/merge

Additional notes

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 4, 2025 •
edited
Loading

Preview links (active after the build_preview check completes)

New or renamed files

Modified Files

@github-actions github-actions bot added the Images Images are added/removed with this PR label Apr 11, 2025
@github-actions github-actions bot added the Guide Content impacting a guide label May 20, 2025
@sabrenner sabrenner requested a review from barieom May 20, 2025 21:16
@sabrenner sabrenner marked this pull request as ready for review May 20, 2025 23:47
@sabrenner sabrenner requested a review from a team as a code owner May 20, 2025 23:47
@buraizu buraizu added the editorial review Waiting on a more in-depth review label May 21, 2025
@buraizu
Copy link
Contributor

buraizu commented May 21, 2025

Created DOCS-10979 for documentation team review

barieom
barieom reviewed May 22, 2025
View reviewed changes
Copy link
Contributor

@barieom barieom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Sam, this is looking great! Can we also add a section with one paragraph under Advanced Tracing? https://docs.datadoghq.com/llm_observability/setup/sdk/python/#advanced-tracing

estherk15
estherk15 reviewed May 23, 2025
View reviewed changes
Copy link
Contributor

@estherk15 estherk15 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much for drafting this! Let me know if you have any questions about my comments or if any of my changes affects the accuracy of your draft.

content/en/llm_observability/trace_proxy_services.md Outdated

## Enabling LLM Observability for a proxy or gateway service

To enable LLM Observability for a proxy or gateway service that might be called from several different ML applications, you can enable LLM Observability without specifying an ML application name. In its place, specify the service name. This can be used to [filter out spans in LLM observability specific to the proxy or gateway service](#observing-llm-gateway-and-proxy-services).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To enable LLM Observability for a proxy or gateway service that might be called from several different ML applications, you can enable LLM Observability without specifying an ML application name. In its place, specify the service name. This can be used to [filter out spans in LLM observability specific to the proxy or gateway service](#observing-llm-gateway-and-proxy-services).
To enable LLM Observability for a proxy or gateway service used by multiple ML applications, you can configure it without specifying an ML application name. Instead, set the service name. This allows you to [filter spans specific to that proxy or gateway service within LLM observability](#observing-llm-gateway-and-proxy-services).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To confirm, are you filtering out spans from that proxy or gateway, or filtering to only show spans from that proxy or gateway?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

filtering to only show spans from that proxy or gateway

estherk15
estherk15 approved these changes May 27, 2025
View reviewed changes
Copy link
Contributor

@estherk15 estherk15 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for considering my suggestions! I left a few more, but they don't block this PR.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 5, 2025 •
edited
Loading

✅ Documentation Team Review

The documentation team has approved this pull request. Thank you for your contribution!

@sabrenner
Copy link
Contributor Author

/merge --delay -@ tomorrow at 9am EST

@dd-devflow
Copy link

dd-devflow bot commented Jun 5, 2025

View all feedbacks in Devflow UI.

2025-06-05 03:33:29 UTC ℹ️ Start processing command /merge --delay -@ tomorrow at 9am EST
If you need support, contact us on Slack #devflow!

2025-06-05 03:33:29 UTC ❌ Devflow: /merge --delay -@ tomorrow at 9am EST

Arguments errors:

  • --delay: value is missing

If you need support, contact us on Slack #devflow with those details!

@sabrenner
Copy link
Contributor Author

/merge --cancel

@dd-devflow
Copy link

dd-devflow bot commented Jun 5, 2025

View all feedbacks in Devflow UI.

2025-06-05 03:33:52 UTC ℹ️ Start processing command /merge --cancel
If you need support, contact us on Slack #devflow!

2025-06-05 03:33:54 UTC ❌ Devflow: /merge --cancel

This merge request is not in the queue and can't be unqueued

To get help about command usage, write /merge --help

If you need support, contact us on Slack #devflow with those details!

@sabrenner
Copy link
Contributor Author

/merge

@dd-devflow
Copy link

dd-devflow bot commented Jun 6, 2025 •
edited
Loading

View all feedbacks in Devflow UI.

2025-06-06 17:00:54 UTC ℹ️ Start processing command /merge

2025-06-06 17:00:58 UTC ℹ️ MergeQueue: queue is disabled

Added to the queue but the mergequeue is not enabled for now.

2025-06-09 16:39:01 UTC ℹ️ MergeQueue: This merge request was already merged

This pull request was merged directly.

@estherk15 estherk15 merged commit 8d2afce into master Jun 9, 2025
19 checks passed
@estherk15 estherk15 deleted the sabrenner/llmobs-proxy-service-quickstart-guide branch June 9, 2025 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@barieom barieom barieom left review comments

@estherk15 estherk15 estherk15 approved these changes

Assignees
No one assigned
Labels
editorial review Waiting on a more in-depth review Guide Content impacting a guide Images Images are added/removed with this PR mergequeue-status: done
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sabrenner @buraizu @barieom @estherk15